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ABSTRACT 


This  paper  and  associated  presentation  describes  an  approach  for  test  and  evaluation  in  a 
net-centric  environment.  A  short  term  effort  is  planned  in  February  to  September  2004  to 
develop  and  apply  appropriate  measures  of  effectiveness  (MOEs),  metrics,  and  methods 
of  data  collection,  analysis  and  evaluation  for  a  representative  test  item’s  performance 
within  a  networked  system  of  systems  environment.  A  net-centric  T&E  approach  is 
prepared  for  the  Rosetta  STONE  Single  Integrated  Picture  Enabling  Technology 
Demonstration  (ETD).  The  Department  of  Defense  has  issued  guidance  and  criteria  in 
the  form  of  joint  concepts,  net-centric  checklists,  and  interoperability  and  supportability 
instructions  for  use  in  program  assessments,  capability  analyses,  and  experimentation. 

The  finding  of  this  research  so  far  are  that  these  criteria,  comprised  of  attributes  derived 
largely  from  network-centric  warfare  concepts  and  commercial  standards,  are  not  yet  in  a 
form  suitable  for  immediate  and  widespread  use  for  test  and  evaluation  (T&E). 

However,  progress  is  being  made.  Status  of  the  effort  will  be  reported  at  the  Symposium 
to  include  lessons  learned  for  planning  future  net-centric  T&E. 

Introduction 

Military  operations  are  increasingly  showing  the  benefits  of  netting  system  of  systems 
together  to  achieve  joint  missions.  Although  not  purchased  originally  to  function  in  an 
interconnected  way,  C3ISR  and  weapon  systems  were  lashed  together  in  Afghanistan  and 
Iraq  to  achieve  incredible  flexibility,  precision  and  speed  in  prosecution  of  time  sensitive 
targets.  New  systems  in  the  acquisition  pipeline  are  increasing  expected  to  be  delivered 
ready  to  plug  and  play  within  the  network  environment.  The  Office  of  the  Secretary  of 
Defense  and  the  Joint  Staff  are  developing  assessment  guidance  and  criteria  for  use  in 
program  assessments,  capability  analyses,  and  experimentation. 

This  paper  describes  and  reports  the  status  of  a  net-centric  T&E  approach  planned  to 
evaluate  the  military  utility  of  the  Rosetta  STONE  Single  Integrated  Picture  Enabling 
Technology  Demonstration  (ETD).  Included  are  a  statement  of  the  problem,  the  concept 
of  operations,  net-centric  requirements  and  measurements,  traditional  T&E  approach,  net- 
centric  approach  for  the  Joint  Military  Utility  Evaluation  (JMUA),  future  applications, 
and  lessons  learned.  Current  status  of  the  effort  will  be  reported  at  the  Symposium. 

Problem 

Warfighter  Problem.  Multiple  after-action  reports,  lessons  learned,  and  Joint  Combat 
Identification  Evaluation  Team  reports  have  consistently  identified  the  necessity  to 
reduce  the  time  sensitive  targeting  decision  time-line.  This  technology  effort  will  address 
these  shortfalls  by  fielding  a  hardware/software  solution  that  the  Warfighter  can  use  to 
resolve  correlation,  fusion,  sensor  registration  and  conduct  of  time  sensitive  targeting. 
Warfighters  are  inundated  with  large  amounts  of  disparate  data  and  information  from 
many  sensors  displayed  on  many  different  displays  which  they  do  not  have  the  time  to 
interpret  and  absorb.  The  diversity  of  sensors  and  data  link  information  makes  it  difficult 
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to  arrive  at  targeting  decisions  in  a  timely  manner.  “Death  by  ones  and  zeros”  is 
preventing  effective  operational  use  of  sensor  advances.  Unfortunately,  existing 
correlation  systems  are  often  limited  to  one  sensor  type.  In  addition,  algorithms  in  current 
systems  exclude  data  to  simplify  the  correlation  and  registration  processes  in  order  to  use 
digital  correlation  engines.  As  a  result,  warfighters  are  unable  to  effectively  leverage  the 
huge  amount  of  available  data  to  provide  the  accuracy  and  identification  that  they  need. 
This  issue  reaches  across  all  services  and  combatant  commands.  The  warfighter  need  is 
to  correlate,  fuse,  and  make  available  all  sources  of  data  to  battle  commanders  and  edge 
users  alike,  in  a  timely  manner  and  provide  that  data  to  in  a  “machine-to-machine”  format 
that  reduces  fratricide  and  enhances  overall  combat  capability. 

Test  &  Evaluation  (T&E)  Problem.  The  Rosetta  STONE  solution  to  address  these 
warfighter  problems  is  end-to-end  data  integration  that  cuts  across  mission  areas, 
platforms,  and  communication  lines.  In  other  words,  it  is  a  networked  solution.  The 
question  to  be  addressed  in  this  paper  is  how  to  verify  that  solution  is  net-centric  and  that 
it  solves  the  problem.  Since  the  traditional  T&E  approach  is  optimized  to  verify 
performance  and  effectiveness  of  point  solutions,  new  criteria  is  needed  to  reflect  the 
realities  of  systems  operating  within  networked  environments.  Such  criteria,  comprised  of 
attributes  derived  largely  from  network-centric  warfare  concepts,  are  just  beginning  to 
emerge  and  not  yet  matured  into  in  a  form  suitable  for  immediate  and  widespread  use  for 
T&E. 

Operational  Overview 

Operational  View.  Joint  tactical  operations  are  the  venues  for  applications  of  Rosetta 
STONE.  As  depicted  in  Figure  1,  this  is  an  environment  where  elements  from  Army, 
Navy,  Air  Force,  and  Marines  Corps  all  interact  with  one  another  as  a  team  to  accomplish 
campaigns,  missions,  and  tasks.  Rosetta  is  software  that  provides  interoperability  at  the 
data  level  among  these  forces.  This  means  that  any  individual  or  unit  can  communicate 
digitally  with  Army,  Air  Force,  Navy,  and  Marines  to  share  information,  coordinate 
movements,  request  fire  support,  confirm  identification,  execute  ordinance  deliveries,  etc. 
Rosetta  provides  the  translation  service  among  various  data  links  so  that  close  air  support 
aircraft  can,  for  example,  “see”  and  avoid  hitting  friendly  ground  units  when  attacking 
targets  in  response  to  a  call  for  air  support  in  a  battle  area.  Over  80  Rosetta-Tactical  Air 
Control  Party  Modernization  (TACP-M)  systems  were  deployed  in  Operation  Iraqi 
Freedom,  including  dismounted  systems,  Air  Support  Operations  Center  (ASOC),  and 
Tactical  Operations  Center  (TOC)  systems.  Soldiers  use  small  ruggedized  laptop  that 
runs  Rosetta.  The  Rosetta  in  TACP  acts  as  a  gateway  and  supports  Air  Force  Application 
Development  Program  (AFAPD)  in  the  F-16,  B-52;  Joint  Variable  Message  Format 
(JVMF)  in  the  F/A-18  &  Army  Tactical  Internet;  and  Fink- 16.  Fikewise,  each  Rosetta 
application  interfacing  the  other  sensors,  data  links,  and  systems  replicate  or  “mirror” 
data  with  all  other  Rosetta  applications  so  that  all  data  is  available  to  all  users  all  the  time. 
In  this  way  the  individual  on  the  ground,  for  example,  can  receive  user  specified  threat 
and  C2  information  from  ISR  platforms,  C2  nodes,  radars,  and  ships  while 
simultaneously  sending  target  information  from  laser  range  finder,  camera,  etc,  directly  to 
fighter  aircraft  for  bomb  delivery.  The  Rosetta  gateway  enables  a  true  machine-to- 
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machine,  sensor-to-shooter  operation.  All  data  resident  on  Rosetta  is  correlated  to 
achieve  a  single  integrated  picture  and  to  determine  identification  of  targets  by  the 
STONE  optical  correlator.  Sensor  data  registration  corrections  are  also  performed  to 
improve  target  location  accuracy  to  weapons  release  grade  levels.  STONE  returns  the 
composite  data  to  Rosetta  so  network  users  can  access  the  integrated  data  and  picture. 


-C'/  :  * 

Operational  View 


Sensor  Data  Posting 


■  Rdsetta 
Mirror 

81 


A  V 


Wideband  Networking 


TACP  Smart  Pirll 
Prioritized  L-16 
AFAPDto  F-16 
JVMFto  F/A-1 8 


JVMF 


Network  Centric  Services 

•  Rosetta  Gateway 

•  STONE  Correlation/Fusion 

•  Battle  Manager 


Figure  1.  Operational  View 

Capabilities  to  be  demonstrated.  The  intent  of  the  Rosetta  STONE  ETD  is  to  improve  the 
decision  making  capability  of  the  Warfighter.  Priorities  are:  1)  improve  targeting 
accuracy  and  reduction  of  Time-Sensitive  Target  (TST)  decision  timelines;  2)  integrate 
of  data  from  disparate  sensors  and  data  links;  3)  register  multi-sensor,  multi-platform 
integration/sensors  in  an  open  architecture;  4)  enhance  coordination  among  shooters  and 
associated  C2  nodes;  5)  integrate  horizontal  information/data  across  platforms;  6)  share 
accurate,  and  relevant  situational  awareness  for  warfighters;  7)  enhance  combat 
identification  of  detected  airbome/ground  objects;  8)  employ  integrated  fire-control 
concepts  in  a  Joint  Fires  Network  environment  using  existing  data  networks;  9)  evaluate 
current  gateway,  correlation,  and  sensor  registration  technologies  used  in  Advanced 
Concept  Technology  Demonstrations  (ACTDs),  JTFWARNET,  FORCENet,  ROBE, 
CAOC-X,  JTRS,  SJFHQ,  DJC2,  and  other  Programs  of  Records. 
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Critical  Operational  Issues  (COI).  Overall  critical  operational  issues  include  how  much 
Rosetta  STONE  applications  demonstrate:  1)  the  ability  to  decrease  engagement  decision 
time  by  enhancing  the  accuracy  of  sensor  data;  2)  the  ability  to  enhance  TST  and  other 
missions’  accuracy  and  precision  by  combining  data  from  disparate  sensors;  3)  the 
migration  and  scalability  of  horizontal  integration  of  networks;  4)  the  ability  to  get 
correlated  track  information  to  the  shooter;  5)  the  ability  to  increase  the  quality  and  speed 
of  distributed  situational  awareness;  and  6)  the  ability  to  perform  gateway  functionalities 
including  correlation,  fusion,  translation,  forwarding,  and  dissemination. 

System  Description 

The  operational  overview  touched  on  what  Rosetta  STONE  can  do.  This  section  will 
describe  the  system  in  more  detail  in  terms  of  its  application  in  the  Navy’s  AEGIS 
weapon  system  (Figure  2).  Rosetta  STONE  is  data  translation,  fusion  and  forwarding 
software  that  can  be  accessed  by  network  users  (network  service)  or  directly  integrated 
with  existing  sensors,  data  links,  and  C2  and  weapon  systems  as  a  dedicated  interface. 

As  such,  it  is  neither  a  platform  nor  a  force  level  capability.  Rather,  it  is  an  integrating 
mechanism  for  interoperability,  which  is  difficult  to  measure  using  traditional 
approaches.  The  Rosetta  communication  link  gateway  software  has  enabled  data 
interoperability  among  military  forces  that  use  different  military  and  commercial  data 
links,  radios,  sensors,  C2  systems,  and  weapons.  The  addition  of  the  STONE  optical 
correlator  offers  to  extract  knowledge  out  of  a  large  variety  of  disparate  data  sources 
currently  managed  by  Rosetta,  plus  several  of  the  major  AEGIS  shipboard  sensors.  This 
combination  will  make  a  single  integrated  picture  available  to  the  crew  on  laptop  or  PC 
anywhere  on  the  ship’s  local  area  network,  and  to  off  board  users  via  data  links  or 
TCP/IP  network  connections.  The  object  under  test  is  a  prototype  configuration  of  the 
STONE  optical  correlator.  Rosetta  prepares  the  sensor  data  and  translates  link  data  for 
processing,  performing  data  normalization  and  pedigree  management  for  the  STONE 
optical  correlation  engine.  Rosetta  processes  data  inputs  and  outputs  using  military  and 
commercial  protocols,  and  message  formats.  The  STONE  correlator  uses  this  variety  of 
processed  sensor  and  link  data  to  rapidly  determine  association  among  targets,  provide 
combat  identification,  and  improve  target  location  accuracy.  Rosetta  distributes  the  data 
using  existing  tactical  communication  links,  or  instantly  accessed  by  users  on  shipboard 
local  area  or  wide  area  networks  as  a  network  service. 
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Figure  2 


Joint  Military  Utility  Assessment 


ACTD  Guidelines.  Rosetta  STONE  ETD  is  sponsored  by  the  ACTD  program  in  the 
Department  of  Defense’s  Office  of  the  Secretary  of  Defense.  Therefore,  a  joint  Military 
Utility  Assessment  (JMUA)  is  planned  in  accordance  with  guidance  provided  for 
ACTDs.  This  guidance  provides  for  a  traditional  T&E  approach  but  allows  more 
flexibility.  Typically,  ACTDs  are  judged  on  their  ability  to  meet  specified  performance 
requirements  and  on  how  useful  it  is  in  the  conduct  of  military  operations.  Since  the 
CJCSI  now  requires  ACTDs  to  include  Net-Ready  requirements,  the  JMUA  approach 
must  be  modified  to  be  more  net-centric. 


Overall  Measures.  The  JMUA  will  address  the  critical  operational  issues  using  measures 
of  performance  (MOPs)  and  measures  of  effectiveness  (MOEs)  to  address  effectiveness 
and  suitability.  MOPs,  technical  characteristics  that  determine  a  particular  aspect  of 
effectiveness  or  suitability,  will  be  used  primarily  to  assess  how  well  the  correlator 
performs.  Measures  of  Effectiveness  (MoEs),  high  level  indicators  of  operational 
effectiveness  or  suitability,  will  be  used  to  assess  operational  utility  within  a  net-centric 
environment.  Many  of  the  critical  operational  issues  will  require  assessments  of  mission 
threads  (e.g.,  time  sensitive  targeting,  air  and  missile  defense,  close  air  support  request, 
etc.)  to  determine  the  level  and  extent  of  horizontal  integration  achieved  across  platforms 
and  nodes.  For  the  Aegis  application,  for  example,  the  JMUA  will  consider  use  of 
common  performance  parameters  derived  from  the  plethora  of  single  integrated  air, 
ground,  and  maritime  pictures.  MOPs  include  quality  of  information  (e.g.,  completeness, 
continuity,  timeliness,  accuracy,  commonality),  level  of  understanding,  degree  of  end-to- 
end  effectiveness,  target  location  accuracy,  time  to  achieve  target  location  accuracy, 
degree  of  combat  identification  achieved,  sensor  registration  accuracy,  degree  of  machine 
to  machine  connectivity,  end-to-end  timeliness,  degree  of  smart  pull  achieved  for  low 
bandwidth  users,  and  number  of  targets  successfully  serviced. 


JMUA  Approach.  United  States  Joint  Forces  Command  (USJFCOM)  has  designated 
JITC  as  the  independent  T&E  lead  for  the  Rosetta  STONE  ETD.  The  JMUA  will  include 
an  assessment  of  the  potential  for  Rosetta  STONE  to  become  both  an  accessible  network 
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service  and  an  embedded  application  to  provide  Joint  Translator/Forwarder  (JxF) 
gateway  functionality  and  replace  multiple  legacy  functions  performing  correlation, 
fusion,  translation,  forwarding,  and  dissemination  throughout  DoD.  JITC,  in  conjunction 
with  participants,  will  plan  and  conduct  interoperability  assessments  of  demonstrated 
Rosetta  STONE  capabilities  in  accordance  with  prescribed  methods  to  provide  certified 
capability  for  technology  transition  into  programs  and  systems  of  record  for  immediate 
warfighter  use.  Demonstrated  capabilities  without  a  Joint  Staff  validated  requirement 
will  be  published  as  an  interoperability  performance  assessment  report. 

General  Test  and  Evaluation  Approach 

Joint  Interoperability  Test  Command’s  (JITC)  Role.  Within  the  DoD,  JITC  has  the  sole 
responsibility  for  certifying  the  Information  Technology  (IT)  systems  &  National 
Security  Systems  (NSS)  for  interoperability  purposes.  NSS  is  a  legal  term  from  the 
Clinger-Cohen  Act  that  includes  DoD  warfighting  military  and  intelligence  systems.  The 
Chairman  of  the  Joint  Chief  of  Staff  Instruction  (CJCSI)  62 12.0 1C  establishes  polices 
and  procedures  for  interoperability  certifications  of  systems  developed  and  delivered 
from  major  programs  in  acquisition  category  (ACAT)  I  and  the  smaller  programs  in 
ACAT  II  through  IV.  The  new  CJCSI  6212.01C  instruction  addresses,  however,  Net 
Ready  Key  Performance  Parameters  as  well  as  expansion  of  testing  and  certification  to 
include  non-ACAT  systems  and  fielded  systems. 

Test  Requirements.  For  the  ACAT  programs,  the  instructions  state  that  the 
interoperability  test  should  be  guided  by  the  Joint  Staff  validated  requirements.  This 
could  come  through  formal  approval  of  the  requirement  documents  such  as  the  Initial 
Capabilities  Document  (ICD),  Capability  Development  Document  (CDD),  or  Capability 
Production  Document  (CPD).  However,  for  non-ACAT  programs  such  as  an  Advanced 
Concept  Technology  Demonstrations  (ACTD),  an  approved  Information  Support  Plan 
(ISP)  would  be  sufficient.  The  Rosetta  STONE  ETD  is  treated  as  an  ACTD  in  terms  of 
T&E  planning. 

Acquisition  Phase.  The  majority  of  all  ACTDs  are  considered  pre-acquisition,  non- 
ACAT  programs.  As  such,  in  order  to  certify  these  systems,  JITC  will  guide  the 
Program  Managers,  Technical  Managers,  and  Operational  Managers  to  develop  an  ISP. 
This  ISP  has  to  be  approved  by  the  sponsoring  organization,  USJFCOM.  For  ACTD 
systems  transitioning  to  ACAT  I-IV  programs,  the  joint  interoperability  certifications  will 
be  issued  before  system  fielding  but  not  before  Milestone  C.  The  ACTD  residuals  or 
“leave-behinds”  in  the  field  must  successfully  pass  an  interoperability  performance 
assessment  conducted  by  JITC. 

Net-Centric  T&E  Approach  vs.  Traditional  T&E  Approach.  The  traditional  T&E 
approach  differs  in  many  ways  with  the  emerging  net-centric  approach.  For  starters,  in  a 
net-centric  approach  the  testers  have  to  be  concerned  with  net-enabled  requirements  in 
addition  to  previous  system  or  systems  of  systems  requirements.  Although  some  of  the 
new  attributes  (such  as  Information  Needs,  Information  Timeliness,  and  Information 
Assurance)  were  part  of  the  traditional  assessment,  these  attributes  have  taken  on  a  new 
definition.  Take  information  assurance  as  an  example.  In  the  past,  testers  were 
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concerned  with  the  information  assurance  attribute  of  the  system  and  its  immediate 
interfaces.  However,  in  a  net-centric  environment,  the  information  assurance  attribute 
depends  on  several  external  parameters,  which  are  beyond  the  control  and  responsibility 
of  the  system  under  test  and  its  connectivity. 

Net-Centric  Requirements  and  Measurements 

Approach  to  a  Network  Centric  T&E  Environment.  This  project  deviates  from  the 
traditional  T&E  approach  by  introducing  net-centric  elements  into  the  criteria.  The 
conceptual  framework  for  network-centric  warfare1  was  used  to  set  the  stage,  which  is 
based  on  the  following  principal  tenets:2 

•  A  robustly  networked  force  improves  information  sharing 

•  Information  sharing  and  collaboration  enhance  quality  of  information  and  shared 
situation  awareness 

•  Shared  situation  awareness  enables  collaboration  and  self-synchronization,  and 
enhances  sustainability  and  speed  of  command 

•  These,  in  turn,  dramatically  increase  mission  effectiveness 

The  question  is  how  to  know  when  we  are  being  successful  in  implementing  net-centric 
capabilities  and  to  what  degree.  The  types  of  measurements  needed  are  not  new  but 
certainly  have  a  different  context,  of  which  the  NCW  conceptual  framework  certainly  is 
one.  Regardless  of  the  framework  used,  the  desirable  metrics  should  be  operational  in 
character.  That  is,  they  should  relate  to  capabilities  but  will  be  quantified  to  reflect 
degrees  of  success  in  specific  systems.  The  metrics  should  also  address  the  critical 
environmental  and  operating  conditions  that  affect  product  performance  and  operational 
utility.  Both  performance  and  operational  utility  will  vary  according  to  specific  operating 
and  stress  conditions  and  these  must  be  carefully  considered  and  incorporated  in  the 
results.  Metrics  will  consider  environmental  conditions  and  legacy  system  performance. 
While  every  system-of-systems  situation  is  dominated  by  numerous  interactions,  it  is 
important  to  focus  on  the  object  under  test  and  address  that  subject  directly  and  its  critical 
interfaces  and  interactions.  This  can  be  accomplished  by  creating  overviews,  both 
operational  and  technical,  of  the  system  and  the  network  within  which  the  system  is 
operating.  These  overviews  are  built  using  the  parameters  described  in  the  Joint 
Technical  Architecture  2.0.  Ultimately,  these  steps  would  lead  to  documenting  a  set  of 
requirements  that  can  be  validated  using  the  joint  staff  approved  process  and  also  used  for 
testing  and  certification.  This  is  a  challenging  but  viable  approach  to  this  problem  and  it 
recognizes  the  dimensions,  complexity  and  structure  of  the  environment  and  the 
contributions  that  many  components  must  make  to  achieve  an  operational  capability.  The 
Office  of  the  Secretary  of  Defense  (OSD)  and  the  Office  of  the  Chairman  of  the  Joint 
Chiefs  of  Staff  (Joint  Staff)  have  been  thinking  a  lot  about  net-centric  attributes,  criteria, 
and  capabilities  and  recently  published  guidance  and  instructions.  A  tailored  T&E 


1  Signori,  David,  Network-Centric  Warfare  Conceptual  Framework, 
http://www.dodccrp.org/events/2002/ncw_workshop/NCWDecWork.htm 

2  http://www.dodccrp.org/research/ncw/ncw.htm 
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construct  must  consider  these  emerging  set  of  universal  net-centric  guidelines  that  are 
affecting  program  management  and  systems  design  for  net-readiness. 

Net-Centric  Attributes  and  Criteria  Checklist.  The  DoD  Chief  Information  Officer 
issued  net-centric  guidance  on  February  24,  2004,  in  the  form  of  Net-Centric  Checklist. 
The  purpose  of  the  checklist  is  to  assist  program  managers  in  understanding  the  net- 
centric  attribute  that  their  programs  need  to  implement  to  move  into  the  net-centric 
environment  in  the  Global  Information  Grid.  The  checklist  is  organized  into  four 
sections:  Data,  Services,  Information  Assurance/Security,  and  Transport.  Although  the 
majority  of  the  checklist  asks  for  specific  technical  descriptions  or  explanations  of  how 
data,  data  services,  data  security,  and  data  transport  are  handled,  there  are  standards 
oriented  design  tenets  that  can  form  the  basis  for  T&E.  Examples  include  the  provision 
of  meta  data  IAW  the  DoD  Discovery  Metadata  Standard  to  make  data  visible; 
mechanisms  for  accessibility  of  data  by  all  potential  consumers;  data  pedigree,  security 
level,  and  access  control  determination;  interoperability  of  data  for  mediation  and 
translation  across  interfaces;  open  architecture  including  web  standards,  XML  standards, 
representational  state  transfer  (REST)  style;  simple  object  access  protocols  (SOAP);  web 
services  description  language  (WSDL);  universal  description,  discovery,  and  integration 
(UDDI)  standard;  web  services  security  (WS-Security)  and  interoperability  (WS-I); 
bandwidth  heterogeneity;  security  authentication  and  protection;  provision  of  Internet 
Protocol  Version  6  (IPv6);  layering;  network  redundancy;  and  network  quality  of  service. 

Net-Ready  Key  Performance  Parameter  Requirements.  The  requirement  for  the  Net- 
Ready  Key  Performance  Parameter  (NR-KPP),  in  lieu  of  the  previous  Interoperability 
KPP  discussed  in  Joint  Staff  documents  CJCSI  3 170.0 1C  and  CJCSM  3170.01,  is 
contained  in  recently  released  CJCSI  62 12.0 1C.3  In  comparison  to  the  DoD  CIO  Net- 
Centric  Checklist  which  is  focused  on  technical  aspects  of  data,  services,  security,  and 
transport,  CJCSI  62 12.0 1C  focuses  on  the  capability  for  systems  to  exchange 
information.  Furthermore,  the  DoD  CIO  checklist  appears  to  be  consistent  with  the  net- 
centric  assessment  criteria  in  Table  F3  of  the  CJCSI  6212. 01C.  Ultimately,  the  NR-KPP 
defines  the  interoperability  requirements  of  the  proposed  system.  The  NR-KPP  assesses 
net-readiness;  information  assurance  requirements;  and  both  the  technical  exchange  of 
information  and  the  end-to-end  operational  effectiveness  of  that  exchange.  Because  the 
Rosetta  STONE  ETD  is  a  pre-acquisition  project,  a  preliminary  NR-KPP  will  be  prepared 
for  the  Rosetta  STONE  ETD  based  on  the  operational  concept,  critical  operational  issues 
that  identify  deficiencies  or  gaps,  and  architectural  views.  This  “draft”  NR-KPP  will  be 
updated  as  the  capability  is  characterized  though  demonstrated  performance.  The 
instruction  also  establishes  policies  and  procedures  for  Joint  Interoperability  Test 
Command  (JITC)  system  interoperability  test  certification.  The  next  section  will 
consider  these  elements  in  the  context  of  a  DoD-wide  assessment  process  of 
interoperability  and  supportability  which  affects  the  T&E  and  JMUA  approach  for  the 
Rosetta  STONE  ETD 


3  Chairman  of  the  Joint  Chiefs  of  Staff  Instruction  6212. 01C,  Interoperability  and  Supportability  of 
Information  Technology  and  National  Security  Systems,  20  November  2003 
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DoD  Interoperability  and  Supportability  Guidance 

DoD  provides  guidelines  for  systems  produced  from  small  projects  including  ACTDs, 
Advanced  Technology  Demonstrations  (ATDs),  Combatant  Commander  Command  and 
Control  Initiative  Program,  and  Combatant  Commander  Field  Assessments  program.  As 
mentioned  previously,  these  smaller  scale  program  activities  are  considered  “Non- 
ACAT”  programs.  As  such,  these  programs  are  required  to  follow  the  Non-ACAT  IT 
and  NSS  Acquisitions  and  Procurements  Process.  DoD  Instruction  4630. 84  outlines  the 
process  and  names  the  responsible  DoD  components  that  must  participate  in  the 
acquisition  cycle  of  the  above  programs.  Figure  3  depicts  the  capability-related, 
outcome-based,  interoperability  and  supportability  process  for  non- AC  AT  IT  and  NSS 
acquisitions  and  procurements. 


Figure  3.  Non-ACAT  IT  and  NSS  Acquisition  and  Procurement  Process5 


4  DoD  Instruction  4630.8,  Procedures  for  Interoperability  and  Supportability  of  Information 
Technology  (IT)  and  National  Security  Systems  (NSS),  ASD(NII),  May  2,  2002 

5DoD  Instruction  4630.8,  Procedures  for  Interoperability  and  Supportability  of  Information 
Technology  (IT)  and  National  Security  Systems  (NSS),  ASD(NII),  draft  revision,  Figure  F 
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Information  Support  Plan.  In  order  to  assess  the  non-ACAT  program,  such  as  the 
Rosetta  STONE  ETD,  an  Information  Support  Plan  (ISP)  must  be  developed.  Among 
other  aspects,  an  ISP  must  document  the  program  needs,  dependencies,  interface 
requirements,  and  incorporate  the  NR-KPP.  The  plan  would  describe  system 
dependencies  and  interface  requirements  in  sufficient  detail  to  enable  testing  and 
verification  of  the  requirements.  The  ISP  would  also  include  the  systems  interface 
descriptions,  infrastructure  and  support  requirements,  standards  profiles,  measures  of 
performance,  and  interoperability  shortfalls.  The  scope  of  the  ISP  would  be  scaled  to  the 
relative  size  and  funding  profile  for  the  program.  Once  an  ISP  has  been  developed,  it 
must  be  reviewed  and  approved  by  the  sponsoring  organization,  USJFCOM.  Any 
unresolved  issues  must  be  reported  to  the  DoD  Chief  Information  Officer. 

Test  Guidance.  There  are  several  documents  that  guide  the  testing  community  in 
analyzing  the  NR-KPP  portion  of  the  ISP.  All  four  pillars  of  the  NR-KPP,  such  as  the 
Net-Centric  Operations  and  Warfare  (NCOW)  Reference  Model,  Integrated  Architecture, 
Key  Interface  Profiles  (KIPs),  and  Information  Assurance,  must  be  addressed.  NCOW 
provides  a  common  language  and  understanding  of  net-centricity  and  specifies  the  core 
capabilities  of  a  net-centric  DoD  architecture.  Integrated  Architecture  defines  the 
Operational  Nodes,  Organizational  Relationships,  Operational  Activity,  Systems 
Functionality  Description,  Systems  Data  Exchange,  and  Technical  Architecture  Profile. 
KIPs  define  organizational  boundaries,  mission  criticality,  capability,  interoperability,  or 
efficiency  issues.  A  KIP  may  affect  multiple  acquisition  programs. 

Key  Interface  Profiles  (KIPs).  The  following  is  a  list  of  17  KIPs  that  are  to  be 
developed.  The  Teleport  KIP  is  the  only  one  that  has  been  completed  so  far.6 

1 .  JTF  to  Coalition 

2.  Logical  Networks  to  DISN  Transport  Backbone 

3.  Space  to  Terrestrial 

4.  TELEPORT 

5.  Client  to  Server 

6.  Application  Server  to  Shared  Data 

7.  DISN  Service  Delivery  Node 

8.  Secure  Enclave  Service  Delivery  Node 

9.  JTF  Component  to  JTF  Headquarters 

10.  Application  Server  to  Database  Server 

11.  Joint  Interconnection  Service 

12.  Management  System  to  Managed  Systems 

13.  Application  to  COE/CCP  (NCES/GES) 

14.  End  System  to  PKI 

15.  Information  Servers  to  IDM  Infrastructure 

16.  IDM  to  Distribution  Infrastructure 

17.  Management  System  to  (integrated)  Management  Systems 


6  Gaetjen,  Tom.  "Interoperability  Process  and  Net-Ready  Key  Performance  Parameter  (vl.O)",  14th  annual 
Interoperability  Conference,  Joint  Interoperability  Test  Command,  Ft.  Huachuca,  18  March  2004. 
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One  of  the  objectives  of  developing  the  KIPs  is  to  define  more  specifically  issues  related 
to  the  interface  and  then  resolve  those  issues.  Once  a  tester  has  identified  which  of  the 
KIPs  is  related  to  the  system  under  testing,  the  specific  parameters  of  that  KIP  are  used  to 
derive  the  requirements  of  the  NR-KPP.  If  the  KIPs  were  available,  those  most  applicable 
to  Rosetta  STONE  ETD  are:  Client  to  Server,  Application  Data  to  Shared  Data,  Secure 
Enclave  Service  Delivery  Node,  Application  Server  to  Database  Server,  and  Application 
to  COE/CPP  (NCES/GES). 

Information  Assurance.  Another  part  of  the  NR-KPP  is  the  Information  Assurance  that 
must  address  availability,  integrity,  authentication,  confidentiality  and  non-repudiation  of 
the  services  that  are  provided  by  the  systems. 

Since  the  KIPs  are  not  yet  completed,  other  guidelines  such  as  the  NCOW,  integrated 
architectures,  CONOPS,  Net-Centric  Attributes  and  Criteria  Checklist,  critical 
operational  issues  will  be  used  to  complete  the  ISP  and  to  specify  detailed  elements  of  the 
NR-KPP. 

Preliminary  Net-Ready  Key  Performance  Parameter.  As  stated  above,  the  ISP  will 
incorporate  the  Net-Ready  Key  Performance  Parameter  (NR-KPP)  of  the  systems.  NR- 
KPPs  will  be  used  to  assess  information  needs,  information  timeliness,  information 
assurance,  and  net-ready  attributes  required  for  both  the  technical  exchange  of 
information  and  the  end-to-end  operational  effectiveness  of  that  exchange  for  a  given 
system.  The  NR-KPP  would  consist  of  verifiable  performance  measures  and  associated 
metrics  required  to  evaluate  the  timely,  accurate,  and  complete  exchange  and  use  of 
information  to  satisfy  the  information  needs  for  a  given  capability.  The  draft  NR-KPP 
for  the  Rosetta  STONE  ETD  would  contain  metrics  for  the  STONE  correlator 
performance,  Rosetta  translator/forwarder  performance,  AEGIS  shipboard  and  off-board 
interfaces,  and  mission  threads  representative  of  end-to-end  integration  to  measure 
military  utility. 

Correlator  Measures  of  Performance.  STONE  correlator  measures  of  performance 
would  include,  for  example,  completeness  (amount  of  truth  targets  included  in  correlator 
output),  timeliness  (data  is  available  when  it  is  needed),  accuracy  (plots,  track  and  radar 
locations  as  reported  or  correlated),  loading  under  a  range  of  scenarios  and  levels  of  data, 
intake  (all  information  used  efficiently),  receiver  operating  characteristic  curve 
(probability  of  correctly  associating  two  track  pairs  vs.  probability  of  mis-associating  two 
track  pairs),  and  number  of  different  types  of  phenomenology  processed  from  sensor  and 
data  links.  As  a  network  service,  the  STONE  correlator  will  be  evaluated  as  a  discovery 
service,  the  measures  of  which  are  still  being  developed. 

Translator/Forwarder  Measures  of  Performance.  Rosetta  will  be  evaluated  in  terms  of 
it  ability  to  correctly  and  completely  translate  tactical  data  link  message  sets  in 
accordance  with  approved  military  standards,  standardization  agreements,  and  forwarding 
rules.  Some  of  the  translator/forwarder  applications  have  been  previously  tested  (Link 
16/Link  11,  L16/JVMF);  others  have  not  (Link  16/CEC,  Link  1 1/CEC). 


12 


Interfaces.  The  interfaces  between  the  sensors  (SLQ-32  EW  system,  SPY-1  radar,  and 
identification  friend  or  foe  (IFF)  systems)  and  data  links  (Fink  16,  Fink  11,  and 
Cooperative  Engagement  Capability  (CEC))  will  be  measured  in  terms  of  the  correctness 
and  completeness  of  the  input  and  output  data.  Rosetta  is  evolving  as  a  network  service 
and  will  be  evaluated  for  net-centric  value-added  features  and  standards  compliance 
based  on  the  attributes  and  criteria  of  the  DoD  CIO  checklist,  net-centric  assessment 
criteria  in  Table  F3  of  the  CJCSI  6212. 01C  ,  NCOW,  available  KIPs,  Joint  Technical 
Architecture,  and  any  other  network-centric  enterprise  service  (NCES)  criteria  What  does 
not  exist  at  the  moment  is  a  comprehensive  list  of  network  service  interface  and 
functional  requirements  needed  for  all  network  services.  These  interfaces  and  associated 
parameters  will  be  analyzed  in  the  Rosetta  STONE  ISP. 

Measures  of  effectiveness.  Based  on  the  network-centric  warfare  conceptual  framework 
and  critical  operational  issues  of  the  Rosetta  STONE  CONOPs,  several  parameters 
appear  important  to  quantify  military  utility,  including  AEGIS  support  of  time  sensitive 
targeting  missions  against  airborne,  surface,  and  ground  targets: 

General  quality  of  information  (e.g.,  completeness,  continuity,  timeliness, 
accuracy,  commonality) 

Degree  of  shared  situation  awareness  (consistency  of  picture  among  variety  of 

users) 

Degree  of  M2M  connectivity,  AKA  scale  of  collaboration  or  extent  of  reach  (%  of 
total  message  types/versions,  %  platforms,  %  C2  nodes) 

Time  sensitive  target  location  accuracy 

Time  sensitive  target  -  time  to  achieve  target  location  accuracy 

Time  sensitive  target  -  %  successful  targeting  delivery  to  shooter 

Degree  of  target  identification  achieved 

Sensor  registration  accuracy 

Degree  of  smart  pull  achieved  for  low  bandwidth  users 

These  and  other  parameters  will  be  analyzed  in  the  information  support  plan  for  use  in 
detailing  the  elements  of  the  draft  net-ready  key  performance  parameter. 

Conclusion 

This  paper  attempted  to  walk  through  an  example  using  a  realistic  case  to  develop  an 
approach  for  net-centric  test  &  evaluation.  The  general  issue  is  how  to  verify  that  a 
proposed  solution  is  net-centric  and  that  it  solves  the  problem.  The  Department  of 
Defense  has  issued  guidance  and  criteria  in  the  form  of  joint  concepts,  net-centric 
checklists,  and  interoperability  and  supportability  instructions  for  use  in  program 
assessments,  capability  analyses,  experimentation,  and  interoperability  testing.  The 
finding  of  this  research  so  far  are  that  these  criteria,  comprised  of  attributes  derived 
largely  from  network-centric  warfare  concepts  and  commercial  standards,  are  not  yet  in  a 
form  suitable  for  immediate  and  widespread  use  for  test  and  evaluation.  Specifically,  the 
detailed  interface  and  environmental  requirements  for  systems  to  successfully  function 
with  and  within  the  global  information  grid  are  not  compiled  in  a  comprehensive  form. 
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Requirements,  the  cornerstone  of  any  net-centric  test  and  evaluation  activity,  are  still 
needed.  While  progress  is  being  made  to  define  these  universal  net-ready  requirements, 
sufficient  criteria  are  available  to  formulate  initial  testing  requirements  for  the  Rosetta 
STONE  enabling  technology  demonstration  both  as  a  hard-wired  system  and  as  a 
network  service.  Such  requirements  will  be  sufficient  to  conduct  an  assessment  which 
can  serve  to  characterize  the  capability  and  inform  future  requirements  analyses  and 
choices.  The  next  step  in  the  process  is  to  develop  an  Information  Support  Plan  to 
document  the  architectures,  interfaces,  and  preliminary  net-ready  key  performance 
parameter. 


Lessons  Learned  for  Net-Centric  T&E  Planning 

Universal  net-centric  requirements  in  the  form  of  net-ready  key  performance  parameters 
and  key  interface  profiles  are  in  development  but  not  yet  mature. 

Until  such  time  as  net-ready  requirements  are  available  for  widespread  use,  the  T&E 
planners  must  tailor  their  approach  based  on  accepted  precedence  and  emerging  criteria. 

Specific  net-centric  requirements  needed  for  Rosetta  STONE  Enabling  Technology 
Demonstration  can  be  developed  by  use  of  an  Information  Support  Plan,  which  can 
further  assess  and  determine  the  details  of  net-ready  key  performance  parameter. 

The  Net-Centric  T&E  approach  needs  a  lot  more  definition  and  will  certainly  create  a  lot 
more  challenges  for  the  testing  communities. 

The  immediate  demand  for  Net-Centric  testing  will  require  an  increased  emphasis  on 
conformance  to  standards. 

There  will  be  more  reliance  on  a  distributed  net-centric  test-bed  infrastructure 

Future  Net-Centric  test  and  evaluation  will  be  more  concern  with  services  rather  than 
systems. 

There  will  be  occasions  when  the  interoperability  assessment  will  deal  with  new  Net- 
Centric  principles  such  as  data  posted  on  the  network  for  immediate  use  before  it  has 
been  processed,  and  only  handling  information  once. 
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Purpose 


■  Develop  an  approach  for  test  and 
evaluation  in  a  net-centric  environment 

■  Use  a  realistic  case  as  an  example: 

Joint  Fires  and  Time  Sensitive  Targeting 
Rosetta  STONE  Single  Integrated  Picture 
Enabling  Technology  Demonstration  (ETD) 


Problem 


■  How  do  you  verify  that  a  proposed  solution 
is  net-centric  and  solves  a  warfighter’s 
problem? 


Wideband  Networking 


JVMF 


Sensor  Data  Posting 


imnin 


Mirror 


TACP  Smart 
Prioritized  L-16 
AFAPDto  F-16 
JVMF  to  F/A-18 


Network  Centric  Services 

•  Rosetta  Gateway 

•  STONE  Correlation/Fusion 

•  Battle  Manager 


Critical  Operational  Issues 


Ability  to  decrease  engagement  decision  time  by  enhancing 
the  accuracy  of  sensor  data 

Ability  to  enhance  TST  and  other  missions’  accuracy  and 
precision  by  combining  data  from  disparate  sensors 

Migration  and  scalability  of  horizontal  integration  of  networks 

Ability  to  get  correlated  track  information  to  the  shooter 

Ability  to  increase  the  quality  and  speed  of  distributed 
situational  awareness 

Ability  to  perform  gateway  functionalities  including  correlation 
fusion,  translation,  forwarding,  and  dissemination. 


System  Descri  ption 


Joint  Military  Utility  Assessment 

■  Can  it  meet  specified  performance 
requirements? 

□  Use  measures  of  performance 

■  Is  it  useful  in  the  conduct  of  military  operations? 

□  Use  measures  of  effectiveness 

■  Is  it  Net-Ready? 

□  Use  measures  of  performance  &  effectiveness 

■  Joint  Forces  Command  Role 

■  Joint  Interoperability  Test  Command  Role 


Emergent  Net-Centric  Technical 
Requirements 

■  DoD  Net-Centric  Data  Strategy 

■  Net-Centric  Data  Visibility:  Tagging  and 
Advertising  Data  Assets  with  Discovery 
Metadata 

■  Net-Centric  Checklist 

□  Data,  Services,  Information  Assurance,  Transport 

■  Net-Centric  Attributes 


Net-Centric  *  Network-Centric 


Net-Centric  Attributes 


Title 

Description 

Metric 

Source 

Internet  Protocol  (IP) 

Data  packets  routed  across  network, 
not  switched  via  dedicated  circuits 

IP  as  the  Convergence  Layer 

Net-Centric  Operations  and  Warfare  Reference 

Model  (NCOW  RM),  Technical  View  compliant 
with  Joint  Technical  Architecture  (JTA) 

NCOW  RM,  GIG  Arch  v2,  IPv6 
Memos  (9  Jun  03  and  29  Sep  03), 

JTA  Memo  23  Nov.  04,  JTA  v6.0 

Secure  and  available 
communications 

Encrypted  initially  for  core  network; 
goal  is  edge-to-edge  encryption  and 
hardened  against  denial  of  service 

Black  Transport  Layer 

Transformational  Communications  Architecture 
(TCA)  compliance;  Technical  View  compliant  with 
JTA 

TCA; 

IA  Component  of  Assured  GIG 
Architecture; 

JTA  Memo  23  Nov.  04,  JTA  v6.0 

Only  handle 
information  once 
(OHIO) 

Data  posted  by  authoritative  sources 
and  visible,  available,  usable  to 
accelerate  decision  making 

Reuse  of  existing  data  repositories 

Community  of  interest  policy  (TBD) 

Post  in  parallel 

Business  process  owners  make  their 
data  available  on  the  net  as  soon  as 
it  is  created 

Data  tagged  and  posted  before  processing 

NCOW  RM, 

Technical  View  compliant  with  JTA 

NCOW  RM,  DoD  Net-Centric  Data 
Strategy  (9  May  03) 

JTA  Memo  23  Nov.  04,  JTA  v6.0 

Smart  pull  (vice  smart 
push) 

Applications  encourage  discovery; 
users  can  pull  data  directly  from  the 
net  or  use  value-added  discovery 
services 

Data  stored  in  public  space  and  advertised 
(tagged)  for  discovery 

NCOW  RM, 

Technical  View  compliant  with  JTA 

NCOW  RM;  DoD  Net-Centric  Data 
Strategy  (9  May  03);  JTA  Memo  23 
Nov.  04,  JTA  v6.0 

Data  centric 

Data  separate  from  applications; 
apps  talk  to  each  other  by  posting 
data 

Metadata  registered  in  DoD  Metadata  Registry 

NCOW  RM, 

Technical  View  compliant  with  JTA 

NCOW  RM;  DoD  Net-Centric  Data 
Strategy  (9  May  03);  JTA  Memo  23 
Nov.  04,  JTA  v6.0 

Application  diversity 

Users  can  pull  multiple  apps  to 
access  same  data  or  choose  same 
app  (e.g.,  for  collaboration) 

Apps  posted  to  net  and  tagged  for  discovery 

NCOW  RM,  Technical  View  compliant  with  JTA 

NCOW  RM;  JTA  Memo  23  Nov.  04, 
JTA  v6.0 

Assured  Sharing 

Trusted  accessibility  to  net 
resources  (data,  services,  apps, 
people,  collaborative  environment, 
etc.) 

Access  assured  for  authorized  users;  denied  for 
unauthorized  users 

Security/IA  policy  (TBD); 

IA  Component  of  Assured  GIG 
Architecture;  JTA  Memo  23  Nov.  04, 
JTA  v6.0 

Quality  of  service 

Data  timeliness,  accuracy, 
completeness,  integrity,  and  ease  of 

use 

Net-ready  key  performance  parameter 

Service  level  agreements  (TBD); 

JTA  Memo  23  Nov.  04,  JTA  v6.0 

DOD  Interoperability  Guidance 
(Technical  +  Operational) 

Joint  Ops,  Functional,  Enabling  Concepts 

Information  Support  Plan 

Net-Ready  Key  Performance  Parameter 

□  Net-Centric  Operations  and  Warfare  (NCOW) 
Reference  Model 

□  Integrated  Architecture 

□  Key  Interface  Profiles  (KIPs) 

□  Information  Assurance 


Measures 

STONE  Corrrelator/Fusion  MOPs 

□  Completeness,  Accuracy,  Loading,  P(false  alarms),  time,  etc 

Rosetta  Translator/Forwarder  MOPs 

Ability  to  correctly  and  completely  translate  tactical  data  link 
message  sets  IAW  specs 

Network  service  and  data  interfaces  -  TBD 

Capability  MOEs  through  TST  mission  threads 

Quality  of  info  (e.g.,  completeness,  continuity,  timeliness,  accuracy) 

□  Degree  of  shared  situation  awareness  (consistency  of  picture  among 
variety  of  users) 

Degree  of  M2M  connectivity,  AKA  scale  of  collaboration  or  extent  of 
reach  (%  of  total  message  types/versions,  %  platforms,  %  C2  nodes) 

□  Time  sensitive  target  location  accuracy  and  time  to  achieve 

□  Quality  of  target  identification  achieved 

□  Time  sensitive  target  -  %  successful  targeting  delivery  to  shooter 
Degree  of  smart  pull  achieved  for  low  bandwidth  users 

□  Time  sensitive  target  -  time  to  detect,  decide,  deliver,  assess 

□  %  successful  time  sensitive  target  missions 


Conclusion 


■  DoD  has  issued  guidance  and  criteria  in  the  form  of  joint  concepts, 
net-centric  checklists,  and  interoperability  and  supportability 
instructions  for  use  in  program  assessments,  capability  analyses, 
experimentation,  and  interoperability  testing 

■  These  criteria,  comprised  of  attributes  derived  largely  from  network¬ 
centric  warfare  concepts  and  commercial  standards,  are  not  yet  in  a 
form  suitable  for  immediate  and  widespread  use  for  test  and 
evaluation 

■  Specifically,  the  detailed  interface  and  environmental  requirements 
for  systems  to  successfully  function  with  and  within  the  global 
information  grid  are  not  compiled  in  a  comprehensive  form 

■  Net-Centric  Requirements  are  evolving  and  are  sufficient  to 
characterize  Rosetta  STONE  as  network  enterprise  services 

■  Need  an  Information  Support  Plan  to  document  the  architectures, 
interfaces,  and  preliminary  net-ready  key  performance  parameter  for 
T&E  planning. 


Lessons  Learned 

■  Until  such  time  as  net-ready  requirements  are  available  for 
widespread  use,  the  T&E  planners  must  tailor  their  approach  based 
on  accepted  precedence  and  emerging  criteria. 

■  Specific  net-centric  requirements  needed  for  Rosetta  STONE 
Enabling  Technology  Demonstration  can  be  developed  by  use  of  an 
Information  Support  Plan,  which  can  further  assess  and  determine 
the  details  of  net-ready  key  performance  parameter. 

■  The  Net-Centric  T&E  approach  needs  a  lot  more  definition  and  will 
certainly  create  a  lot  more  challenges  for  the  testing  communities. 

■  The  immediate  demand  for  Net-Centric  testing  will  require  an 
increased  emphasis  on  conformance  to  standards. 

■  There  will  be  more  reliance  on  a  distributed  net-centric  test-bed 
infrastructure 

■  Future  Net-Centric  test  and  evaluation  will  be  more  concern  with 
services  rather  than  systems. 

■  Future  interoperability  assessments  will  deal  with  new  Net-Centric 
attributes  such  as  data  posted  on  the  network  for  immediate  use 
before  it  has  been  processed,  and  only  handling  information  once 


Questions? 


Reference  Sources: 

http://www.horizontalfusion.dod.mil/fy05/ref  docs.html 

http://www.dtic.mil/iointvision/ 

http://www.dod.mil/nii/doc/ 
http://www.dtic.mil/cics  directives/ 


